UCI Adult Income Dataset - Exploratory and Descriptive Analysis

Author

Affiliation

NIWENSHUTI Adeline

Junior Data Analyst

Published

June 25, 2025

In this notebook, we carry out an in-depth exploratory and descriptive analysis of the UCI Adult Income Dataset, a widely used dataset for income prediction tasks based on individual demographic and employment attributes.

This phase of analysis is essential for uncovering patterns, detecting potential biases, and gaining intuition about the dataset’s structure before applying any modelling procedures. We examine the distribution of key numerical and categorical variables, investigate relationships between demographic features and income levels, and use visualizations to summarize insights. Particular focus is placed on income disparities across age groups, geographical regions, races, and education-occupation combinations, helping lay a solid foundation for downstream modeling and policy-relevant interpretation.

We begin our analysis by importing the core Python libraries required for data handling, numerical computation, visualization, and directory management:

pandas: Enables efficient manipulation, filtering, and aggregation of structured tabular data, forming the backbone of our analysis pipeline.
numpy: Provides support for fast numerical operations, array-based computation, and statistical routines.
os: Facilitates interaction with the file system, allowing us to construct flexible and portable directory paths for data and output management.
plotly.express: A high-level graphing library that enables the creation of interactive, publication-quality visualizations, which we use extensively to uncover patterns and present insights throughouutputs for reproducibility.

Code

# import libraries
import pandas as pd
import numpy as np
import os
import plotly.express as px

Define and Create Directory Paths

To ensure reproducibility and organized storage, we programmatically create directories if they don’t already exist for:

raw data
processed data
results
documentation

These directories will store intermediate and final outputs for reproducibility.

Loading the Cleaned Dataset

We load the cleaned version of the UCI Adult Income Dataset from the processed data directory into a Pandas DataFrame. The head(10) function shows the first ten records, giving a glimpse into the data columns such as age, workclass, education_num, etc.

Code

adult_data_filename = os.path.join(processed_dir,'adult_cleaned.csv')
adult_df = pd.read_csv(adult_data_filename)
adult_df.head(10)

	age	workclass	fnwgt	education_num	marital_status	relationship	race	sex	capital_gain	hours_per_week	income	education_level	occupation_grouped	native_region	age_group
0	39	government	77516	13	single	single	white	male	2174	40	<=50k	tertiary	white collar	north_america	36-45
1	50	self-employed	83311	13	married	male spouse	white	male	0	13	<=50k	tertiary	white collar	north_america	46-60
2	38	private	215646	9	divorced or separated	single	white	male	0	40	<=50k	secondary-school graduate	blue collar	north_america	36-45
3	53	private	234721	7	married	male spouse	black	male	0	40	<=50k	secondary	blue collar	north_america	46-60
4	28	private	338409	13	married	female spouse	black	female	0	40	<=50k	tertiary	white collar	central_america	26-35
5	37	private	284582	14	married	female spouse	white	female	0	40	<=50k	tertiary	white collar	north_america	36-45
6	49	private	160187	5	divorced or separated	single	black	female	0	16	<=50k	secondary	service	central_america	46-60
7	52	self-employed	209642	9	married	male spouse	white	male	0	45	>50k	secondary-school graduate	white collar	north_america	46-60
8	31	private	45781	14	single	single	white	female	14084	50	>50k	tertiary	white collar	north_america	26-35
9	42	private	159449	13	married	male spouse	white	male	5178	40	>50k	tertiary	white collar	north_america	36-45

Dataset Dimensions and Data Types

Here, we examine the structure of the dataset:

There are 32,513 entries and 16 variables.
The dataset includes both numerical (e.g., age, hours_per_week) and categorical variables (e.g., sex, education_level).

Understanding data types and null entries is essential before proceeding with analysis.

Code

summary_df = pd.DataFrame({'Column':adult_df.columns,
                            'Data Type':adult_df.dtypes.values,
                           'Missing Values':adult_df.isnull().sum().values
})
summary_df

Table 1: Overview of dataset columns, their data types, and the count of missing values in each column.

	Column	Data Type
0	age	int64
1	workclass	object
2	fnwgt	int64
3	education_num	int64
4	marital_status	object
5	relationship	object
6	race	object
7	sex	object
8	capital_gain	int64
9	capital_loss	int64
10	hours_per_week	int64
11	income	object
12	education_level	object
13	occupation_grouped	object
14	native_region	object
15	age_group	object

Summary Statistics: Numerical Variablesecessity.

Table 2: Summary statistics for numerical variables in the dataset, including count, mean, standard deviation, min, and quartile values.

	age	fnwgt	education_num	capital_gain	capital_loss	hours_per_week
count	32513.000000	3.251300e+04	32513.000000	32513.000000	32513.000000	32513.000000
mean	38.590256	1.897942e+05	10.081629	1079.239812	87.432719	40.440962
std	13.638932	1.055788e+05	2.572015	7390.625650	403.243596	12.350184
min	17.000000	1.228500e+04	1.000000	0.000000	0.000000	1.000000
25%	28.000000	1.178330e+05	9.000000	0.000000	0.000000	40.000000
50%	37.000000	1.783560e+05	10.000000	0.000000	0.000000	40.000000
75%	48.000000	2.370510e+05	12.000000	0.000000	0.000000	45.000000
max	90.000000	1.484705e+06	16.000000	99999.000000	4356.000000	99.000000

This summary provides a snapshot of key distribution characteristics. We see that:

Age ranges from 17 to 90, with a mean of 38.6 years. It is slightly right-skewed (positively skewed). While the average age is approximately 38.6 years, an examination of the percentiles reveals that the majority of individuals are clustered in the younger to middle-age range, with fewer observations in the older age brackets. This skewed age distribution might suggest labor force participation is concentrated in specific age groups, which could reflect broader demographic or economic realities.
Capital gains/losses are highly skewed, with most values at 0 (the 75th percentile is 0). This indicates that a small number of individuals report very large gains or losses, especially evident in the capital gain variable which reaches up to $99,999. These variables act as proxies for wealth-related income that goes beyond regular wages or salaries. Individuals with non-zero values for capital gains or losses often represent a distinct socioeconomic subset of the population — typically more financially literate, or with access to investment assets. The stark inequality in their distributions mirrors real-world disparities in asset ownership and investment returns.
The dataset has individuals working anywhere from 1 to 99 hours per week, with a median of 40. This aligns with the standard full-time work week in many countries (8 hours per day for 5 working days). The mean is slightly above that at 40.4 hours, suggesting a mild right skew, with a small subset of individuals working significantly longer hours. The mode is also 40, further reinforcing the prevalence of full-time work. A non-trivial number of individuals report working very few hours, possibly due to part-time work, unemployment, or semi-retirement. On the other extreme, some report working more than 45 hours per week, which may indicate multiple jobs, weekend-work, self-employment, or informal labor, and could reflect socioeconomic necessity.

Summary Statistics: Categorical Variables focus on the working-age population.

Code

adult_df['workclass'].value_counts(normalize=True).rename_axis('unique values').reset_index(name='proportion')

Table 3: Distribution of the workclass variable showing the proportion of each unique category within the dataset.

	unique values	proportion
0	private	0.696644
1	government	0.133793
2	self-employed	0.112447
3	unknown	0.056470
4	voluntary	0.000431
5	unemployed	0.000215

The private sector dominates, employing ~69.7% of the population. The government sector (13.4%) and self-employment (11.2%) also make up substantial portions of the workforce. A small fraction is labeled as “unknown” (5.6%), which may correspond to missing or ambiguous data entries. Tiny proportions are voluntary (0.04%) or unemployed (0.02%), possibly underreported or underrepresented in the sample.

marital_status

Code

adult_df['marital_status'].value_counts(normalize=True).rename_axis('unique values').reset_index(name='proportion')

Table 4: Proportion of each category in the marital_status variable.

	unique values	proportion
0	married	0.460862
1	single	0.327684
2	divorced or separated	0.180912
3	widowed	0.030542

Married individuals make up the largest group (46.1%), followed by those who are single (32.8%) and divorced or separated (18.1%). Widowed individuals represent a small minority (~3.1%).

relationship

Code

adult_df['relationship'].value_counts(normalize=True).rename_axis('unique values').reset_index(name='proportion')

Table 5: Distribution of the relationship variable by category proportions.

	unique values	proportion
0	male spouse	0.405315
1	single	0.360686
2	child	0.155599
3	female spouse	0.048227
4	extended relative	0.030173

The majority are labeled as “male spouse” (40.5%) or “single” (36.1%). Smaller categories include children (15.6%), female spouses (4.8%), and extended relatives (3.0%). The dominance of male spouse reflects the dataset’s gendered structure and may point to traditional family roles. The relative scarcity of “female spouse” roles suggests potential gender imbalances in how income-earning is reported within households.

race

Code

adult_df['race'].value_counts(normalize=True).rename_axis('unique values').reset_index(name='proportion')

Table 6: Proportional breakdown of the race variable categories.

	unique values	proportion
0	white	0.854151
1	black	0.096023
2	asian or pacific islander	0.031926
3	american indian or eskimo	0.009565
4	other	0.008335

The dataset is overwhelmingly composed of White individuals (~85.4%). Other racial groups include Black (9.6%), Asian or Pacific Islander (3.2%), American Indian or Eskimo (1.0%), and Other (0.8%). The racial imbalance limits the generalizability of models trained on this data. Smaller racial groups may suffer from limited statistical power, affecting fairness and performance in predictive modeling.

sex

Code

adult_df['sex'].value_counts(normalize=True).rename_axis('unique values').reset_index(name='proportion')

Table 7: Proportional distribution of the sex variable within the dataset.

	unique values	proportion
0	male	0.669209
1	female	0.330791

Males constitute 66.9% of the dataset, with females making up the remaining 33.1%. This male-skewed distribution could be due to sampling (e.g., primary earners in households), workforce participation patterns, or reporting biases.

education_level

Code

adult_df['education_level'].value_counts(normalize=True).rename_axis('unique values').reset_index(name='proportion')

Table 8: Distribution of educational attainment levels (education_level) by proportion.

	unique values	proportion
0	secondary-school graduate	0.322456
1	tertiary	0.247809
2	some-college	0.223787
3	secondary	0.093932
4	associate	0.075324
5	primary	0.035155
6	preschool	0.001538

Secondary-school graduates form the largest educational group (~32%), highlighting the central role of high school completion in the labor force. Tertiary education holders — those with university or equivalent degrees — account for nearly 25% of the population, representing a substantial segment with advanced qualifications. A notable 22.4% have attended some college without necessarily earning a degree, suggesting that partial post-secondary education is common, yet may not always translate into formal certification. The remaining 20% are distributed among those with only secondary education (9.4%), associate degrees (7.5%), primary school (3.5%), and a very small group with only preschool education (0.15%). It is ecident that the education distribution is skewed toward mid- to high-level education, with relatively few individuals having only basic schooling. This reflects a dataset that largely captures working-age adults in formal labor, which may underrepresent the least-educated populations.

occupation_grouped

Code

adult_df['occupation_grouped'].value_counts(normalize=True).rename_axis('unique values').reset_index(name='proportion')

Table 9: Proportion of each occupational category (occupation_grouped) in the dataset.

	unique values	proportion
0	white collar	0.508474
1	blue collar	0.308861
2	service	0.125704
3	unknown	0.056685
4	military	0.000277

White-collar occupations are the most prevalent (~51%), followed by blue-collar, service, and unknown. Smaller categories include military, which is marginal. Essentially, slightly over half of individuals in the dataset work in professional, managerial, sales, clerical, or tech-support roles. This suggests the dataset is heavily weighted toward professional and administrative occupations. Nearly a third of the population works in manual labor or skilled trade positions (craft, transport, machine operation, farming, etc.). This indicates a significant segment engaged in physically intensive or technical labor.

native_region

Code

adult_df['native_region'].value_counts(normalize=True).rename_axis('unique values').reset_index(name='proportion')

Table 10: Distribution of native_region categories by proportion.

	unique values	proportion
0	north_america	0.923261
1	asia	0.020638
2	other	0.017870
3	central_america	0.016117
4	europe	0.016024
5	south_america	0.006090

The vast majority of individuals are from North America (~92.3%). Smaller proportions are from Central America, Asia, Europe, South America, and a generic Other category. The heavy concentration of North American individuals reflects the U.S. focus of the dataset.

age_group

Code

adult_df['age_group'].value_counts(normalize=True).rename_axis('unique values').reset_index(name='proportion')

Table 11: Proportional breakdown of the age_group categories in the dataset.

	unique values	proportion
0	26-35	0.261465
1	36-45	0.246086
2	46-60	0.224156
3	18-25	0.167533
4	61-75	0.064313
5	<18	0.029065
6	76+	0.007382

The largest groups are 26–35 and 36–45, followed by 46–60. These three age groups represent about 73% of the dataset. Very few individuals are under 18 or above 75, consistent with the dataset’s focus on the working-age population.

Income Distribution

Given that income is the target variable, most of the analysis hereafter will be based on it. We first of all examine the income distribution in the dataset.

Code

fig = px.pie(
    adult_df_income,
    names='income',
    values='total',
    title='Overall Income Distribution',
    color_discrete_sequence=px.colors.sequential.RdBu
)

fig.update_layout(
    template="presentation",
    legend_title=dict(text='Income Level'),
    paper_bgcolor="rgba(0, 0, 0, 0)",
    plot_bgcolor="rgba(0, 0, 0, 0)",
    width=500,     # reduced width
    height=400     # reduced height
)

fig.show()

fig.write_image(os.path.join(results_dir, 'income_distribution_pie_chart.png'))
fig.write_html(os.path.join(results_dir, 'income_distribution_pie_chart.html'))

This pie chart visualizes the overall income split: 76% of individuals earn ≤50K, while 24% earn >50K. This means that nearly 3 out of 4 individuals fall into the lower income bracket (<=50K). This shows that there is a significant imbalance.

Income by Age Group

Code

import plotly.express as px
import os

fig = px.bar(
    adult_df_income_age, 
    x='age_group',
    y='percentage',
    color='income',
    title='Income Distribution by Age Group',
    barmode='group',
    height=350,  # Reduced height
    width=600,   # Optional: you can adjust this as needed
    color_discrete_sequence=px.colors.sequential.RdBu,
    text='percentage'
)

fig.update_traces(
    texttemplate='%{text:.2f}%', 
    textposition='outside'
)

fig.update_layout(
    template="presentation",
    xaxis_title='Age Group',
    yaxis_title='Percentage of Population',
    legend_title_text='Income Level',
    paper_bgcolor="rgba(0,0,0,0)",
    plot_bgcolor="rgba(0,0,0,0)",
    margin=dict(t=50, b=50, l=40, r=40)  # Optional: tighter margins
)

fig.show()

fig.write_image(os.path.join(results_dir, 'income_distribution_by_agegroup_bar_plot.png'))
fig.write_html(os.path.join(results_dir, 'income_distribution_by_agegroup_bar_plot.html'))

The bar chart visualizes the income distribution across age groups, using percentages within each group. There is an evident pattern in terms of income progression over the years with a gradual increase in terms of the number of people earning >50K starting from 0 amongst those aged 18 and below, peaking between 36 and 60 years, then declining after 60 years but not to zero.

All individuals under 18 earn <=50K, likely due to being students, minors, or ineligible for full-time employment. Extremely few young adults (2.1%) exceed 50K, as most are early in their careers, pursuing education, or in entry-level jobs. For the 26-35 age group, there’s a noticeable improvement — roughly 1 in 5 individuals in this group earn >50K, reflecting early career progression and accumulation of qualifications/experience. A substantial income increase is seen in the 36-45 age group: over a third now earn >50K. This is typically considered prime earning age where individuals settle into stable, higher-paying positions. Highest proportion of >50K earners is seen amongst individuals aged between 46 and 60— nearly 4 in 10. This reflects career maturity, peak seniority levels, and accumulated experience. There’s a drop-off in high incomes as many transition to retirement, part-time, or less demanding roles in the age group 61-75. Yet about 1 in 4 still earn >50K. Most in 76+ age group earn <=50K, likely due to retirement, pensions, or fixed incomes — but a small minority still earn higher incomes, possibly through continued work or investments.

Income by Native region

Code

fig = px.bar(
    adult_df_income_native_region,
    x='native_region',
    y='percentage',
    color='income',
    title='Income Distribution by Native Region (%)',
    barmode='group',
    color_discrete_sequence=px.colors.sequential.RdBu,
    text='percentage', 
    width=800,    # reduced from 1000
    height=500    # reduced from 1100
)

fig.update_traces(
    texttemplate='%{text:.2f}%', 
    textposition='outside'
)

fig.update_layout(
    template="presentation",
    xaxis_title='Native Region',
    yaxis_title='Percentage',
    legend_title_text='Income Level',
    paper_bgcolor="rgba(0,0,0,0)",
    plot_bgcolor="rgba(0,0,0,0)",
    margin=dict(t=60, b=60, l=40, r=40)  # optional tighter margins
)

fig.show()

fig.write_image(os.path.join(results_dir, 'income_distribution_by_native_region_bar_plot.png'))
fig.write_html(os.path.join(results_dir, 'income_distribution_by_native_region_bar_plot.html'))

Asia (30.7%) and Europe (29.2%) have the highest proportions of high-income earners. This suggests these immigrant groups might be better integrated into high-paying professional roles, or may represent a more skilled migrant profile in the dataset. Central America (11.1%) and South America (12.1%) have the lowest proportions of >50K earners. With 24.2% of North Americans earning >50K, this serves as a middle-ground baseline. Interestingly, both Asian and European groups outperform the native-born population proportionally in high-income brackets. The ‘Other’ group sits around 25.1%, close to North America’s rate. This likely reflects a diverse mix of regions not explicitly listed.

Asian or Pacific Islander (26.6%) and White (25.6%) populations have the highest proportions of >50K earners. Asians/Pacific Islanders marginally outperform Whites, a pattern often attributed to occupational concentration in high-paying sectors like technology and medicine. On the other hand, American Indian or Eskimo (11.6%), Black (12.4%), and Other (9.2%) groups show significantly lower rates of high-income earners. These figures reflect long-standing economic disparities rooted in historical exclusion, occupational segregation, and systemic inequality.

The stark differences in high-income proportions:

Between Whites and Blacks: 25.6% vs 12.4% — slightly over double the proportion.
Between Asians and Others: 26.6% vs 9.2% — nearly triple.

These disparities are consistent with well-documented wage gaps and underrepresentation of marginalized groups in higher-paying roles.

Income by Race

Code

fig = px.bar(
    adult_df_income_race,
    x='race',
    y='percentage',
    color='income',
    title='Income Distribution by Race (%)',
    barmode='group',
    color_discrete_sequence=px.colors.sequential.RdBu,
    text='percentage',
    width=700,       # reduced from 900
    height=400       # reduced from 1000
)

fig.update_traces(
    texttemplate='%{text:.2f}%', 
    textposition='outside'
)

fig.update_layout(
    template="presentation",
    xaxis_title='Race',
    yaxis_title='Percentage',
    legend_title_text='Income Level',
    paper_bgcolor="rgba(0,0,0,0)",
    plot_bgcolor="rgba(0,0,0,0)",
    margin=dict(t=50, b=50, l=40, r=40)
)

fig.show()

fig.write_image(os.path.join(results_dir, 'income_distribution_by_race_bar_plot.png'))
fig.write_html(os.path.join(results_dir, 'income_distribution_by_race_bar_plot.html'))

Code

adult_df_income_edu_occ.head(15)

	education_level	occupation_grouped	income	total	edu_occ
29	secondary-school graduate	blue collar	<=50k	3976	secondary-school graduate\|blue collar
56	tertiary	white collar	>50k	3545	tertiary\|white collar
55	tertiary	white collar	<=50k	3369	tertiary\|white collar
45	some-college	white collar	<=50k	3003	some-college\|white collar
36	secondary-school graduate	white collar	<=50k	2900	secondary-school graduate\|white collar
38	some-college	blue collar	<=50k	1503	some-college\|blue collar
32	secondary-school graduate	service	<=50k	1444	secondary-school graduate\|service
20	secondary	blue collar	<=50k	1349	secondary\|blue collar
6	associate	white collar	<=50k	1015	associate\|white collar
41	some-college	service	<=50k	902	some-college\|service
46	some-college	white collar	>50k	858	some-college\|white collar
30	secondary-school graduate	blue collar	>50k	796	secondary-school graduate\|blue collar
37	secondary-school graduate	white collar	>50k	731	secondary-school graduate\|white collar
23	secondary	service	<=50k	663	secondary\|service
12	primary	blue collar	<=50k	634	primary\|blue collar

From the bar chart, we can pick out the largest groups per income-level. We see that secondary-school graduates working a blue collar job occupy the largest group in the dataset (3976). This reflects a common socio-economic profile: individuals with basic schooling in manual or technical trades predominantly earning lower incomes. The largest high-income group are tertiary-educated individuals in white collar roles. This highlights the strong earning advantage conferred by higher education and skilled jobs.

Some of the key patterns we can get from the dataset are:

Education matters, but isn’t deterministic

Tertiary education combined with white-collar work offers the highest income prospects. Yet a substantial number of tertiary-educated white-collar workers earn <=50K, likely early career, part-time, or structural pay gaps.

Blue-collar and service work predominantly pay <=50K, regardless of education.

Even some college education doesn’t guarantee high incomes in these sectors. Manual and service sector income is highly occupation-dependent (some skilled trades can break the 50K mark).

Some non-tertiary education groups do reach >50K

Secondary-school graduates in blue-collar and white-collar work have decent representation among >50K earners. This reflects upward mobility possible through skilled trades, tenure, or niche roles.

--- title: UCI Adult Income Dataset - Exploratory and Descriptive Analysis author: - name: NIWENSHUTI Adeline affiliation: Junior Data Analyst date: '2025-06-25' format: html: page-layout: full self-contained: true code-fold: true code-tools: true toc-location: left toc-title: contents jupyter: kernelspec: display_name: Python [python] * language: python name: python3 --- In this notebook, we carry out an in-depth exploratory and descriptive analysis of the UCI Adult Income Dataset, a widely used dataset for income prediction tasks based on individual demographic and employment attributes. This phase of analysis is essential for uncovering patterns, detecting potential biases, and gaining intuition about the dataset’s structure before applying any modelling procedures. We examine the distribution of key numerical and categorical variables, investigate relationships between demographic features and income levels, and use visualizations to summarize insights. Particular focus is placed on income disparities across **age groups**, **geographical regions**, **races**, and **education-occupation combinations**, helping lay a solid foundation for downstream modeling and policy-relevant interpretation. We begin our analysis by importing the core Python libraries required for **data handling**, **numerical computation**, **visualization**, and **directory management**: - `pandas`: Enables efficient manipulation, filtering, and aggregation of structured tabular data, forming the backbone of our analysis pipeline. - `numpy`: Provides support for fast numerical operations, array-based computation, and statistical routines. - `os`: Facilitates interaction with the file system, allowing us to construct flexible and portable directory paths for data and output management. - `plotly.express`: A high-level graphing library that enables the creation of interactive, publication-quality visualizations, which we use extensively to uncover patterns and present insights throughouutputs for reproducibility. ```{python} # import libraries import pandas as pd import numpy as np import os import plotly.express as px ``` ## Define and Create Directory Paths To ensure reproducibility and organized storage, we programmatically create directories if they don't already exist for: - **raw data** - **processed data** - **results** - **documentation** These directories will store intermediate and final outputs for reproducibility. ```{python} #| echo: false # get working directory Current_dir = os.getcwd() # on one directory up to the root directory project_root_dir = os.path.dirname(Current_dir) # define paths to the data folders data_dir = os.path.join(project_root_dir,'data') raw_dir = os.path.join(data_dir,'raw') processed_dir = os.path.join(data_dir,'processed') # define paths to result folder results_dir = os.path.join(project_root_dir,'results') # define paths to docs folder docs_dir = os.path.join(project_root_dir,'docs') # create directories if they do not eFileExists os.makedirs(raw_dir, exist_ok = True) os.makedirs(processed_dir, exist_ok = True) os.makedirs(results_dir, exist_ok = True) os.makedirs(docs_dir, exist_ok = True) ``` ## Loading the Cleaned Dataset We load the cleaned version of the UCI Adult Income Dataset from the processed data directory into a Pandas DataFrame. The `head(10)` function shows the first ten records, giving a glimpse into the data columns such as `age`, `workclass`, `education_num`, etc. ::: {.panel-tabset} ## code ```{python} #| output: false adult_data_filename = os.path.join(processed_dir,'adult_cleaned.csv') adult_df = pd.read_csv(adult_data_filename) adult_df.head(10) ``` ## output ```{python} #| echo: false adult_data_filename = os.path.join(processed_dir,'adult_cleaned.csv') adult_df = pd.read_csv(adult_data_filename) adult_df.head(10) ``` ::: ## Dataset Dimensions and Data Types Here, we examine the structure of the dataset: - There are *32,513* entries and *16* variables. - The dataset includes both **numerical** (e.g., `age`, `hours_per_week`) and **categorical** variables (e.g., `sex`, `education_level`). Understanding data types and null entries is essential before proceeding with analysis. ```{python} #| echo: false #| output: false adult_df.shape ``` ```{python} #| echo: false #| output: false adult_df.info() ``` ```{python} #| label: tbl-dataset-overview #| tbl-cap: Overview of dataset columns, their data types, and the count of missing values in each column. summary_df = pd.DataFrame({'Column':adult_df.columns, 'Data Type':adult_df.dtypes.values, 'Missing Values':adult_df.isnull().sum().values }) summary_df ``` ## Summary Statistics: Numerical Variablesecessity. ```{python} #| label: tbl-summary-numerical #| tbl-cap: Summary statistics for numerical variables in the dataset, including count, mean, standard deviation, min, and quartile values. #| echo: false adult_df.describe() ``` This summary provides a snapshot of key distribution characteristics. We see that: - Age ranges from 17 to 90, with a mean of 38.6 years. It is slightly right-skewed (positively skewed). While the average age is approximately 38.6 years, an examination of the percentiles reveals that the majority of individuals are clustered in the younger to middle-age range, with fewer observations in the older age brackets. This skewed age distribution might suggest labor force participation is concentrated in specific age groups, which could reflect broader demographic or economic realities. - Capital gains/losses are highly skewed, with most values at 0 (the 75th percentile is 0). This indicates that a small number of individuals report very large gains or losses, especially evident in the capital gain variable which reaches up to $99,999. These variables act as proxies for wealth-related income that goes beyond regular wages or salaries. Individuals with non-zero values for capital gains or losses often represent a distinct socioeconomic subset of the population — typically more financially literate, or with access to investment assets. The stark inequality in their distributions mirrors real-world disparities in asset ownership and investment returns. - The dataset has individuals working anywhere from 1 to 99 hours per week, with a median of 40. This aligns with the standard full-time work week in many countries (8 hours per day for 5 working days). The mean is slightly above that at 40.4 hours, suggesting a mild right skew, with a small subset of individuals working significantly longer hours. The mode is also 40, further reinforcing the prevalence of full-time work. A non-trivial number of individuals report working very few hours, possibly due to part-time work, unemployment, or semi-retirement. On the other extreme, some report working more than 45 hours per week, which may indicate multiple jobs, weekend-work, self-employment, or informal labor, and could reflect socioeconomic necessity. ## Summary Statistics: Categorical Variables focus on the working-age population. ```{python} #| label: tbl-summary-categorical #| tbl-cap: Summary statistics for categorical variables in the dataset, including the number of unique values, most frequent category (top), and its frequency (freq). #| echo: false #| output: false adult_df.describe(include='object') ``` ```{python} #| label: tbl-workclass-distribution #| tbl-cap: Distribution of the `workclass` variable showing the proportion of each unique category within the dataset. adult_df['workclass'].value_counts(normalize=True).rename_axis('unique values').reset_index(name='proportion') ``` The private sector dominates, employing ~69.7% of the population. The government sector (13.4%) and self-employment (11.2%) also make up substantial portions of the workforce. A small fraction is labeled as "unknown" (5.6%), which may correspond to missing or ambiguous data entries. Tiny proportions are voluntary (0.04%) or unemployed (0.02%), possibly underreported or underrepresented in the sample. `marital_status` ```{python} #| label: tbl-marital-status-distribution #| tbl-cap: Proportion of each category in the `marital_status` variable. adult_df['marital_status'].value_counts(normalize=True).rename_axis('unique values').reset_index(name='proportion') ``` Married individuals make up the largest group (46.1%), followed by those who are single (32.8%) and divorced or separated (18.1%). Widowed individuals represent a small minority (~3.1%). `relationship` ```{python} #| label: tbl-relationship-distribution #| tbl-cap: Distribution of the `relationship` variable by category proportions. adult_df['relationship'].value_counts(normalize=True).rename_axis('unique values').reset_index(name='proportion') ``` The majority are labeled as "male spouse" (40.5%) or "single" (36.1%). Smaller categories include children (15.6%), female spouses (4.8%), and extended relatives (3.0%). The dominance of `male spouse` reflects the dataset’s gendered structure and may point to traditional family roles. The relative scarcity of "female spouse" roles suggests potential gender imbalances in how income-earning is reported within households. `race` ```{python} #| label: tbl-race-distribution #| tbl-cap: Proportional breakdown of the `race` variable categories. adult_df['race'].value_counts(normalize=True).rename_axis('unique values').reset_index(name='proportion') ``` The dataset is overwhelmingly composed of White individuals (~85.4%). Other racial groups include Black (9.6%), Asian or Pacific Islander (3.2%), American Indian or Eskimo (1.0%), and Other (0.8%). The racial imbalance limits the generalizability of models trained on this data. Smaller racial groups may suffer from limited statistical power, affecting fairness and performance in predictive modeling. `sex` ```{python} #| label: tbl-sex-distribution #| tbl-cap: Proportional distribution of the `sex` variable within the dataset. adult_df['sex'].value_counts(normalize=True).rename_axis('unique values').reset_index(name='proportion') ``` Males constitute 66.9% of the dataset, with females making up the remaining 33.1%. This male-skewed distribution could be due to sampling (e.g., primary earners in households), workforce participation patterns, or reporting biases. `education_level` ```{python} #| label: tbl-education-level-distribution #| tbl-cap: Distribution of educational attainment levels (`education_level`) by proportion. adult_df['education_level'].value_counts(normalize=True).rename_axis('unique values').reset_index(name='proportion') ``` Secondary-school graduates form the largest educational group (~32%), highlighting the central role of high school completion in the labor force. Tertiary education holders — those with university or equivalent degrees — account for nearly 25% of the population, representing a substantial segment with advanced qualifications. A notable 22.4% have attended some college without necessarily earning a degree, suggesting that partial post-secondary education is common, yet may not always translate into formal certification. The remaining 20% are distributed among those with only secondary education (9.4%), associate degrees (7.5%), primary school (3.5%), and a very small group with only preschool education (0.15%). It is ecident that the education distribution is skewed toward mid- to high-level education, with relatively few individuals having only basic schooling. This reflects a dataset that largely captures working-age adults in formal labor, which may underrepresent the least-educated populations. `occupation_grouped` ```{python} #| label: tbl-occupation-grouped-distribution #| tbl-cap: Proportion of each occupational category (`occupation_grouped`) in the dataset. adult_df['occupation_grouped'].value_counts(normalize=True).rename_axis('unique values').reset_index(name='proportion') ``` White-collar occupations are the most prevalent (~51%), followed by blue-collar, service, and unknown. Smaller categories include military, which is marginal. Essentially, slightly over half of individuals in the dataset work in professional, managerial, sales, clerical, or tech-support roles. This suggests the dataset is heavily weighted toward professional and administrative occupations. Nearly a third of the population works in manual labor or skilled trade positions (craft, transport, machine operation, farming, etc.). This indicates a significant segment engaged in physically intensive or technical labor. `native_region` ```{python} #| label: tbl-native-region-distribution #| tbl-cap: Distribution of `native_region` categories by proportion. adult_df['native_region'].value_counts(normalize=True).rename_axis('unique values').reset_index(name='proportion') ``` The vast majority of individuals are from North America (~92.3%). Smaller proportions are from Central America, Asia, Europe, South America, and a generic Other category. The heavy concentration of North American individuals reflects the U.S. focus of the dataset. `age_group` ```{python} #| label: tbl-age-group-distribution #| tbl-cap: Proportional breakdown of the `age_group` categories in the dataset. adult_df['age_group'].value_counts(normalize=True).rename_axis('unique values').reset_index(name='proportion') ``` The largest groups are 26–35 and 36–45, followed by 46–60. These three age groups represent about 73% of the dataset. Very few individuals are under 18 or above 75, consistent with the dataset's focus on the working-age population. ## Income Distribution Given that `income` is the target variable, most of the analysis hereafter will be based on it. We first of all examine the income distribution in the dataset. ```{python} #| echo: false #| output: false adult_df_income = adult_df.groupby('income').size().reset_index(name='total') adult_df_income ``` ```{python} fig = px.pie( adult_df_income, names='income', values='total', title='Overall Income Distribution', color_discrete_sequence=px.colors.sequential.RdBu ) fig.update_layout( template="presentation", legend_title=dict(text='Income Level'), paper_bgcolor="rgba(0, 0, 0, 0)", plot_bgcolor="rgba(0, 0, 0, 0)", width=500, # reduced width height=400 # reduced height ) fig.show() fig.write_image(os.path.join(results_dir, 'income_distribution_pie_chart.png')) fig.write_html(os.path.join(results_dir, 'income_distribution_pie_chart.html')) ``` This pie chart visualizes the overall income split: 76% of individuals earn ≤50K, while 24% earn >50K. This means that nearly 3 out of 4 individuals fall into the lower income bracket (<=50K). This shows that there is a significant imbalance. ### Income by Age Group ```{python} #| echo: false #| output: false adult_df_income_age= adult_df.groupby(['age_group','income']).size().reset_index(name='total_by_age').sort_values(['age_group','income'],ascending=True) adult_df_income_age ``` ```{python} #| echo: false #| output: false total_per_group = adult_df_income_age.groupby('age_group')['total_by_age'].transform('sum') adult_df_income_age['percentage'] = (adult_df_income_age['total_by_age'] / total_per_group * 100).round(2) adult_df_income_age['percentage'] ``` ```{python} #| echo: false #| output: false total_per_group = adult_df_income_age.groupby('age_group')['total_by_age'].transform('sum') adult_df_income_age['percentage'] = (adult_df_income_age['total_by_age'] / total_per_group * 100).round(2) adult_df_income_age['percentage'] ``` ```{python} import plotly.express as px import os fig = px.bar( adult_df_income_age, x='age_group', y='percentage', color='income', title='Income Distribution by Age Group', barmode='group', height=350, # Reduced height width=600, # Optional: you can adjust this as needed color_discrete_sequence=px.colors.sequential.RdBu, text='percentage' ) fig.update_traces( texttemplate='%{text:.2f}%', textposition='outside' ) fig.update_layout( template="presentation", xaxis_title='Age Group', yaxis_title='Percentage of Population', legend_title_text='Income Level', paper_bgcolor="rgba(0,0,0,0)", plot_bgcolor="rgba(0,0,0,0)", margin=dict(t=50, b=50, l=40, r=40) # Optional: tighter margins ) fig.show() fig.write_image(os.path.join(results_dir, 'income_distribution_by_agegroup_bar_plot.png')) fig.write_html(os.path.join(results_dir, 'income_distribution_by_agegroup_bar_plot.html')) ``` The bar chart visualizes the income distribution across age groups, using percentages within each group. There is an evident pattern in terms of income progression over the years with a gradual increase in terms of the number of people earning >50K starting from 0 amongst those aged 18 and below, peaking between 36 and 60 years, then declining after 60 years but not to zero. All individuals under 18 earn <=50K, likely due to being students, minors, or ineligible for full-time employment. Extremely few young adults (2.1%) exceed 50K, as most are early in their careers, pursuing education, or in entry-level jobs. For the 26-35 age group, there's a noticeable improvement — roughly 1 in 5 individuals in this group earn >50K, reflecting early career progression and accumulation of qualifications/experience. A substantial income increase is seen in the 36-45 age group: over a third now earn >50K. This is typically considered prime earning age where individuals settle into stable, higher-paying positions. Highest proportion of >50K earners is seen amongst individuals aged between 46 and 60— nearly 4 in 10. This reflects career maturity, peak seniority levels, and accumulated experience. There's a drop-off in high incomes as many transition to retirement, part-time, or less demanding roles in the age group 61-75. Yet about 1 in 4 still earn >50K. Most in 76+ age group earn <=50K, likely due to retirement, pensions, or fixed incomes — but a small minority still earn higher incomes, possibly through continued work or investments. ### Income by Native region ```{python} #| echo: false #| output: false adult_df_income_native_region = adult_df.groupby(['native_region', 'income']).size().reset_index(name='total_income_distr').sort_values('total_income_distr',ascending = False) adult_df_income_native_region ``` ```{python} #| echo: false #| output: false total_per_region = adult_df_income_native_region.groupby('native_region')['total_income_distr'].transform('sum') adult_df_income_native_region['percentage'] = (adult_df_income_native_region['total_income_distr']/total_per_region) * 100 adult_df_income_native_region ``` ```{python} fig = px.bar( adult_df_income_native_region, x='native_region', y='percentage', color='income', title='Income Distribution by Native Region (%)', barmode='group', color_discrete_sequence=px.colors.sequential.RdBu, text='percentage', width=800, # reduced from 1000 height=500 # reduced from 1100 ) fig.update_traces( texttemplate='%{text:.2f}%', textposition='outside' ) fig.update_layout( template="presentation", xaxis_title='Native Region', yaxis_title='Percentage', legend_title_text='Income Level', paper_bgcolor="rgba(0,0,0,0)", plot_bgcolor="rgba(0,0,0,0)", margin=dict(t=60, b=60, l=40, r=40) # optional tighter margins ) fig.show() fig.write_image(os.path.join(results_dir, 'income_distribution_by_native_region_bar_plot.png')) fig.write_html(os.path.join(results_dir, 'income_distribution_by_native_region_bar_plot.html')) ``` Asia (30.7%) and Europe (29.2%) have the highest proportions of high-income earners. This suggests these immigrant groups might be better integrated into high-paying professional roles, or may represent a more skilled migrant profile in the dataset. Central America (11.1%) and South America (12.1%) have the lowest proportions of >50K earners. With 24.2% of North Americans earning >50K, this serves as a middle-ground baseline. Interestingly, both Asian and European groups outperform the native-born population proportionally in high-income brackets. The 'Other' group sits around 25.1%, close to North America’s rate. This likely reflects a diverse mix of regions not explicitly listed. Asian or Pacific Islander (26.6%) and White (25.6%) populations have the highest proportions of >50K earners. Asians/Pacific Islanders marginally outperform Whites, a pattern often attributed to occupational concentration in high-paying sectors like technology and medicine. On the other hand, American Indian or Eskimo (11.6%), Black (12.4%), and Other (9.2%) groups show significantly lower rates of high-income earners. These figures reflect long-standing economic disparities rooted in historical exclusion, occupational segregation, and systemic inequality. The stark differences in high-income proportions: - **Between Whites and Blacks**: 25.6% vs 12.4% — slightly over double the proportion. - **Between Asians and Others**: 26.6% vs 9.2% — nearly triple. These disparities are consistent with well-documented wage gaps and underrepresentation of marginalized groups in higher-paying roles. ### Income by Race ```{python} #| echo: false #| output: false adult_df_income_race = adult_df.groupby(['race', 'income']).size().reset_index(name='total_income_distr').sort_values('total_income_distr',ascending = False) adult_df_income_race ``` ```{python} #| echo: false #| output: false total_per_race = adult_df_income_race.groupby('race')['total_income_distr'].transform('sum') adult_df_income_race['percentage'] = (adult_df_income_race['total_income_distr']/total_per_race) * 100 adult_df_income_race ``` ```{python} fig = px.bar( adult_df_income_race, x='race', y='percentage', color='income', title='Income Distribution by Race (%)', barmode='group', color_discrete_sequence=px.colors.sequential.RdBu, text='percentage', width=700, # reduced from 900 height=400 # reduced from 1000 ) fig.update_traces( texttemplate='%{text:.2f}%', textposition='outside' ) fig.update_layout( template="presentation", xaxis_title='Race', yaxis_title='Percentage', legend_title_text='Income Level', paper_bgcolor="rgba(0,0,0,0)", plot_bgcolor="rgba(0,0,0,0)", margin=dict(t=50, b=50, l=40, r=40) ) fig.show() fig.write_image(os.path.join(results_dir, 'income_distribution_by_race_bar_plot.png')) fig.write_html(os.path.join(results_dir, 'income_distribution_by_race_bar_plot.html')) ``` ```{python} #| echo: false #| output: false adult_df_income_edu_occ = adult_df.groupby(['education_level', 'occupation_grouped','income']).size().reset_index(name='total').sort_values('total',ascending = False) adult_df_income_edu_occ ``` ```{python} #| echo: false #| output: false adult_df_income_edu_occ['edu_occ']=(adult_df_income_edu_occ['education_level']+"|")+adult_df_income_edu_occ['occupation_grouped'] adult_df_income_edu_occ ``` ```{python} adult_df_income_edu_occ.head(15) ``` ```{python} #| echo: false #| output: true num = 15 adult_df_combos = adult_df_income_edu_occ.head(num) fig = px.bar( adult_df_combos, x = 'total', y = 'edu_occ', color = 'income', orientation = 'h', title = f'Top {num} Education and Occupation Groups Combinations by Income Group', # barmode = 'group', height = 700, width=1100, color_discrete_sequence=px.colors.sequential.RdBu, text = 'total' ) fig.update_layout(template="presentation", xaxis_title='Number of Individuals', yaxis_title='Education | Occupation Group', legend_title=dict(text='Income Level'), margin=dict(l=450, r=50, t=50, b=50), paper_bgcolor = "rgba(0, 0, 0, 0)", plot_bgcolor = "rgba(0, 0, 0, 0)") fig.update_traces(textposition='inside') fig.show() fig.write_image(os.path.join(results_dir, 'income_distribution_by_eduandocc_bar_plot.png')) fig.write_html(os.path.join(results_dir, 'income_distribution_by_eduandocc_bar_plot.html')) ``` From the bar chart, we can pick out the largest groups per income-level. We see that secondary-school graduates working a blue collar job occupy the largest group in the dataset (3976). This reflects a common socio-economic profile: individuals with basic schooling in manual or technical trades predominantly earning lower incomes. The largest high-income group are tertiary-educated individuals in white collar roles. This highlights the strong earning advantage conferred by higher education and skilled jobs. Some of the key patterns we can get from the dataset are: - **Education matters, but isn’t deterministic** Tertiary education combined with white-collar work offers the highest income prospects. Yet a substantial number of tertiary-educated white-collar workers earn <=50K, likely early career, part-time, or structural pay gaps. - **Blue-collar and service work predominantly pay <=50K, regardless of education.** Even some college education doesn’t guarantee high incomes in these sectors. Manual and service sector income is highly occupation-dependent (some skilled trades can break the 50K mark). - **Some non-tertiary education groups do reach >50K** Secondary-school graduates in blue-collar and white-collar work have decent representation among >50K earners. This reflects upward mobility possible through skilled trades, tenure, or niche roles.